Not an explainer — a build session. You design a distributed, partitioned, replicated log system from a blank page, one decision at a time, for your own wealth/mutual-fund platform's scale. Every segment starts with a callout telling you exactly what to draw or decide before you're allowed to look at the reference design.
How to use this: Every segment opens with a dashed "build it yourself first" box. Stop there — write, sketch, or say your answer out loud before clicking reveal. The reveal is a reference design from a staff-engineer design review, not "the" answer. If your design differs but you can defend the tradeoff, you're doing this right.
step 01
Functional & non-functional requirements
You're not designing "a Kafka." You're designing the log system that sits under mutual fund order flow, IPO allotment, NSDL demat sync, and MOFSL/PolicyBazaar integrations across 80+ microservices. Requirements first — everything downstream will be judged against these numbers.
✎ build it yourself first
Write your own FR list (5–7 bullets) and NFR targets with real numbers — throughput, p99 latency, your consistency/availability trade-off — for this log system inside your domain. Then write one line on what you're explicitly scoping OUT, and why leaving it out is itself a design decision.
R1Reference: what would a staff engineer write for FRs and NFRs here?reveal
Functional requirements
Producers (order-service, nsdl-adapter, mofsl-webhook, nav-ingest) append records to named, partitioned topics.
Records within a partition are delivered to consumers in the exact order they were appended.
Multiple independent consumer groups (settlement, reporting, audit, notification) read the same topic at their own pace, without affecting each other.
A consumer can resume from its last committed position after a crash or restart — no manual replay coordination needed.
A new consumer, onboarded months later, can replay history within the retention window.
Topic and partition metadata (who's leader, what's the replica set) is discoverable by every client without a single point of failure.
Writes are acknowledged only once durably replicated to a configurable quorum — not just written to one disk.
Non-functional targets
Throughput: sustain 3.5k msgs/sec peak (your NAV cutoff window), steady-state 400–600/sec, headroom to 3x for surge.
Latency: p99 produce-to-durable-ack under 50ms; p99 end-to-end (produce to consumer read) under 500ms for real-time flows.
Durability > availability for order/settlement topics: a partition with no in-sync replica should refuse writes rather than silently accept and risk loss.
Why this matters: Kafka itself doesn't have one durability/availability setting — it's per-topic (acks, min.insync.replicas). Writing this out first forces you to notice that "the system" doesn't have one NFR profile; your topics do. That single realization drives almost every later decision.
R2What should you explicitly scope OUT — and why does that matter as much as what's in scope?reveal
Scope out: stream processing / joins / windowed aggregation (that's a layer built on this system, e.g. Kafka Streams/Flink — not the log itself), schema enforcement (that's a registry sitting beside the broker, not inside it), and exactly-once cross-topic transactions (a significant added protocol on top of the base replication model).
Why this matters: Every one of those is a legitimate, common ask — and every one of them, if you try to build it into segment 1, will silently bend your component boundaries in segment 3 before you've even validated the core log works. Scoping out isn't laziness — it's protecting the shape of the core design from feature creep disguised as a requirement.
step 02
Scope & the core design decision
One or two decisions here will shape every segment after this. Get them wrong and you'll be fighting your own architecture by segment 6.
✎ build it yourself first
Commit, in writing, to: (1) your ordering guarantee scope — global or per-partition-key? (2) your delivery semantics — at-least-once, at-most-once, or exactly-once, and where the cost of that choice lands. Justify both against your own segment-1 numbers before revealing.
R3Reference decision — ordering scope and delivery semantics.reveal
Ordering: per-partition-key ordering only. No global order across a topic. Order for a given folio/account is guaranteed by routing every event for that key to the same partition.
Delivery: at-least-once by default, everywhere. Exactly-once only for the narrow slice that truly can't tolerate a duplicate (settlement writes) — implemented as idempotent writes at the consumer, not as a broker-wide guarantee.
Why this matters: Global ordering requires a single writer for the whole topic, which caps your throughput at one partition's worth of I/O — directly violating your segment-1 throughput target. And broker-wide exactly-once (transactional producers + read-committed consumers everywhere) triples your write-path latency for topics that never needed it, like UI click events. Committing to "per-key order, at-least-once by default" isn't a compromise — it's the decision that makes segment-1's 3.5k msgs/sec and sub-50ms ack even possible.
R4If a future team demands exactly-once end-to-end for a new topic, what do you tell them?reveal
You don't change the broker's default guarantee for everyone. You offer it as an opt-in mode per topic: idempotent producer (sequence numbers dedupe retries) plus a transactional write path for that specific producer-consumer pair. It costs that topic latency and throughput — a cost the rest of the platform shouldn't pay for a need it doesn't have.
Why this matters: The moment you make one team's requirement the platform default, every future topic inherits a cost it never asked for. Per-topic opt-in keeps the platform's baseline fast and lets the rare expensive requirement pay for itself.
step 03
Component breakdown
Draw your own boxes before you see the reference. The interesting question isn't what the components are — it's why some of them are not allowed to be the same component.
✎ build it yourself first
Sketch 5–7 boxes for your system. Then answer: why can't the hot-path append logic live in the same component as the durable replication/sync path? Write your reasoning before revealing.
7 boxes: append path, replication manager, fetch/read path, log segment storage, controller/metadata quorum, consumer group coordinator, and the client libraries (producer/consumer) that talk to all of the above.
Why the append path and replication manager can't be one component: the append path's only job is "write this record to my local log as fast as possible and hand back an offset" — it's a single-writer, sequential-disk-write hot path measured in microseconds. The replication manager's job is "coordinate with N other machines over the network and wait for acknowledgment" — a fundamentally different latency class (milliseconds, subject to network partitions). Fusing them means every local append blocks on network I/O, which caps your throughput at your slowest follower's network latency — exactly what segment 1's sub-50ms target can't survive. Separating them lets the leader append locally, return fast, and let replication catch up asynchronously up to the acks contract.
step 04
Low-level API design
Concrete endpoints, not prose. For each: who actually calls it — a service, an operator, an autoscaler? — and what happens if it's called twice.
✎ build it yourself first
Write 5–6 API cards of your own: method/path, purpose, caller, payload, idempotency behavior. Then decide: should authorization be scoped by team ownership, or by blast radius? Write your answer before revealing.
R6Reference API cards — data plane and control plane.reveal
RPCProduce(topic, partition_key, payload, acks)
Purpose
Append a record to a partition, chosen by hashing partition_key.
Caller
Any producer service — order-service, nsdl-adapter, nav-ingest-job.
Payload
{topic, key, value, headers, acks: 0|1|all}
Idempotency
Producer attaches a sequence number per (producer_id, partition); broker dedupes retries with the same sequence — safe to retry blindly.
Not idempotent — calling twice with different partition counts must be rejected, not merged, or you silently break key-ordering guarantees for existing data.
ADMINDescribeCluster() / Metadata(topics)
Purpose
Discover current leader/replica assignment for topics — every client library calls this before Produce/Fetch and caches it.
Caller
Producer and consumer clients, plus operator tooling and monitoring.
Payload
{topics: [...]}
Idempotency
Pure read — trivially idempotent.
Why blast radius, not ownership: "Who owns this API" tells you who to page. "What's the blast radius if this call is misused" tells you how to scope the token. Produce scoped by ownership means any order-service credential can produce to any topic that team's key happens to also have — but scoped by blast radius, a settlement-topic write credential is deliberately narrower than a low-stakes-topic write credential, because a bad write to mf.order.events costs far more than a bad write to a UI-click topic. CreateTopic and DescribeCluster are asymmetric for the same reason: read access is cheap to grant broadly; the ability to reshape partition layout is not.
step 05
Data model & sizing
Sketch your own schema and estimate real numbers for your platform's scale before revealing.
✎ build it yourself first
Sketch what's actually stored on disk per partition, and estimate: how many topics, partitions, and bytes/day across your 80+ microservices? Then answer the gut-check: is this actually a "big data" storage problem, or not? Decide what's durable config vs. ephemeral in-memory derived state, and why mixing them up would be a mistake.
R7Reference data model and sizing.reveal
On disk, per partition: a sequence of immutable log segment files (e.g. 1GB each), each with a paired sparse offset index (offset → byte position) and a timestamp index. Segments roll over by size or time; old segments are deleted or compacted per the topic's retention policy.
partition-3/
00000000000000000000.log <- segment (records, append-only)
00000000000000000000.index <- offset -> byte-position lookup
00000000000000000000.timeindex
00000000000045210000.log <- next segment after rollover
...
Sizing (illustrative, across the wealth platform):
Topics
~40 active
Total partitions
~420 cluster-wide
Peak write rate
3.5k msgs/sec
Avg record size
~800 bytes
Daily volume
~190 GB/day pre-replication
Replicated footprint
~570 GB/day (RF=3)
Durable vs. ephemeral: log segments, indexes, and committed offsets are durable — written to disk, replicated, survive a restart. Broker-side metadata caches (which broker thinks it's leader for which partition, right now), consumer group in-memory assignment state, and ISR membership snapshots are derived/ephemeral — rebuilt from the controller/quorum on restart, never treated as a source of truth.
Why conflating them is a mistake: if you accidentally treat an in-memory ISR snapshot as durable truth, a broker restart can "forget" a valid replica was in sync and either falsely exclude it (unnecessary re-catch-up) or falsely trust a stale one (data loss risk). The controller/quorum is the only durable source of who's really in sync — everything else is a fast, disposable cache of that fact.
The big-data gut check: ~570 GB/day sounds large, but at 7-day retention that's ~4TB steady-state per replicated topic set — comfortably single-digit-node storage, not a big-data problem. The actual hard part isn't volume, it's throughput + ordering + durability under concurrent access — a systems problem, not a "shard it across 200 nodes" problem. Don't let the word "Kafka" trick you into over-building for scale you don't have yet.
step 06
Core algorithm: the append + ISR replication protocol
Pick one mechanism and get concrete. This is the single piece of logic your entire durability story depends on.
✎ build it yourself first
Write actual pseudocode — not prose — for: (1) the leader's append operation, (2) how a follower catches up and joins the ISR, (3) how the leader decides when to ack a producer. Then answer: what happens if the ISR shrinks to zero in-sync replicas mid-write?
// Leader-side state per partition
struct PartitionState {
log: AppendOnlyLog // local durable segment writer
isr: Set<ReplicaId> // currently in-sync replicas, includes self
high_watermark: Offset // highest offset acked by ALL of isr
pending_acks: Map<Offset, List<Waiter>>
lock: SingleWriterLock // one append at a time, this partition only
}
// 1. Append (hot path — must stay fast)
function append(record, acks_mode):
lock.acquire()
offset = log.write(record) // sequential disk write, local only
lock.release()
if acks_mode == 0:
return ack_immediately() // fire and forget
elif acks_mode == 1:
return ack_immediately() // leader-local durability only
else: // acks == "all"
pending_acks[offset].add(current_waiter)
return wait_until(offset <= high_watermark) // blocks until ISR catches up
// 2. Follower replication loop (async, off the hot path)
function follower_replicate_loop(follower):
while true:
fetch_offset = follower.last_replicated_offset + 1
records = leader.log.read_from(fetch_offset)
follower.log.write(records)
follower.last_replicated_offset += len(records)
leader.update_follower_progress(follower.id, follower.last_replicated_offset)
// 3. High watermark advance + ISR membership (leader-side, periodic)
function recompute_high_watermark_and_isr():
for replica in isr:
if replica.lag_ms() > replica.lag.threshold_ms:
isr.remove(replica) // fell behind -> out of ISR, catch-up continues separately
high_watermark = min(replica.last_replicated_offset for replica in isr)
release_waiters_up_to(high_watermark) // unblocks pending acks==all producers
Why the append and the ISR recompute are separate functions, not one: the append path needs to stay a single fast local write with a lock held for microseconds. If ISR bookkeeping (network round-trip checks, lag calculation) happened inside that same lock, every append would serialize behind the slowest follower's heartbeat — reintroducing the exact coupling segment 3 said to avoid.
R9The ISR shrinks to zero in-sync replicas (leader included, hypothetically) mid-write. What happens?reveal
With min.insync.replicas set (e.g. 2) and acks=all, the leader must refuse the write and return a "not enough replicas" error the moment ISR count drops below the configured minimum — it does not silently downgrade to a weaker guarantee. The producer is responsible for retry/backoff until ISR recovers.
Why this matters: the alternative — quietly accepting the write with fewer in-sync copies than promised — is how "the broker said it was durable" becomes "the broker lied," which is the single worst thing a log-as-source-of-truth can do for a settlement pipeline. Refusing to write is a worse user experience in the moment and a strictly better guarantee for the business. This is the concrete mechanism behind segment 1's "durability over availability" NFR for order/settlement topics.
step 07
Supporting subsystem: the consumer group coordinator
Every distributed log needs something deciding "who reads what" as consumers join, leave, and crash. Design its loop precisely.
✎ build it yourself first
Design the coordinator's heartbeat/timeout/rebalance loop as precisely as you can — when does it trigger, what does it do. Then decide: should this coordinator be one centralized process per group, or a distributed protocol among the consumers themselves? Justify before revealing.
R10Reference: coordinator loop and the centralized-vs-distributed call.reveal
// Runs on one broker elected as coordinator for this group_id
function coordinator_loop(group):
loop every heartbeat_check_interval:
for member in group.members:
if now() - member.last_heartbeat > member.session_timeout_ms:
group.members.remove(member)
trigger_rebalance(group) // member presumed dead
on receive JoinGroup(member_id):
group.members.add(member_id)
trigger_rebalance(group) // new member -> reassign
on receive Heartbeat(member_id):
group.members[member_id].last_heartbeat = now()
function trigger_rebalance(group):
group.state = REBALANCING
partitions = all_partitions(group.subscribed_topics)
assignment = partition_assignor(partitions, group.members) // e.g. sticky/round-robin
broadcast_assignment(assignment)
group.state = STABLE
Centralized, per group. One broker acts as coordinator for a given group_id (chosen deterministically, e.g. by hashing the group ID to a partition of an internal topic) — not a fully distributed consensus protocol among consumers themselves.
Why centralized wins here: a distributed protocol among consumer instances (each independently deciding partition ownership via gossip or voting) trades one coordinator bottleneck for a harder problem — network-partition-tolerant distributed consensus, for a decision that's made infrequently (only on join/leave/crash) and needs to be fast to converge, not fast to sustain high QPS. A single coordinator broker handles this fine because rebalances are rare events, not steady-state traffic — the coordinator never sits on the hot data path. The real risk isn't the centralization, it's a rebalance storm (too many join/leave events too close together) — which is a tuning problem (session timeout, static membership), not an architecture problem.
step 08
Failure handling & state machine
Pull every state you've mentioned so far — partition, replica, consumer group — into one diagram you draw yourself.
✎ build it yourself first
Draw the state machine: partition states, replica states, consumer group states, and the transitions between them. Then find the "stuck forever" risk — is there any state your design could enter and never leave without a timeout forcing an exit? Write your answer before revealing.
R11Reference state machine.reveal
PARTITION (per broker's view of a partition)
[Offline] --controller assigns--> [Becoming-Leader] --logs recovered--> [Leader]
[Offline] --controller assigns--> [Follower] --catches up--> [In-Sync Follower]
[Leader]/[Follower] --broker unreachable--> [Offline] --> controller re-elects
REPLICA (per follower, from leader's view)
[Catching-Up] --lag below threshold--> [In-Sync (ISR member)]
[In-Sync] --lag exceeds threshold--> [Catching-Up] --removed from ISR--
[Catching-Up] --replica never catches up (timeout)--> [Considered Dead]
CONSUMER GROUP
[Stable] --member join/leave/crash--> [Rebalancing]
[Rebalancing] --new assignment broadcast + acked--> [Stable]
[Rebalancing] --rebalance timeout exceeded--> [Rebalancing] (retry, with backoff)
The "stuck forever" risk: without a hard ceiling, [Rebalancing] can theoretically loop if members keep joining/leaving faster than assignment can stabilize (a rebalance storm), and [Becoming-Leader] can stall if log recovery hangs on a corrupted segment.
Why every state needs a timeout-forced exit: a distributed system state machine that only transitions on "success" events (join succeeded, catch-up succeeded, recovery succeeded) has no way out of a state where the success event never comes. The fix in both cases is the same shape: a bounded timeout that forces a transition regardless of outcome — [Becoming-Leader] times out into "mark partition unavailable, alert operator" rather than hanging forever, and [Rebalancing] uses exponential backoff plus a max-rebalance-attempts ceiling that eventually kicks a flapping member out of the group entirely rather than blocking the whole group's progress indefinitely. Retry and idempotency plug in exactly here: every one of these forced exits is safe only because the operations being retried (fetch, produce with sequence numbers, offset commit) are idempotent — a forced retry after a timeout never double-applies an effect.
step 09
Scaling this component itself
Not the systems this log serves — this log system. How do more brokers and a bigger controller stay coordinated without becoming the bottleneck.
✎ build it yourself first
Decide: as broker count grows, what keeps controller/metadata decisions from becoming a coordination bottleneck? Then write one concrete metric, tied back to your segment-1 NFRs, that would tell you "it's time to scale out."
R12Reference: keeping the controller/metadata layer from becoming the bottleneck.reveal
Two moves: (1) the controller itself runs as a small quorum (3 or 5 nodes, Raft-style — this is what KRaft replaced ZooKeeper with) rather than one node, so it's fault-tolerant without being "everyone votes on everything." (2) metadata changes are batched and propagated asynchronously to brokers as an event log of their own (a metadata topic), not synchronously pushed on every single change — brokers pull/cache metadata and only re-fetch on a version bump, so a metadata update doesn't mean N broker-to-controller round trips per partition change.
Why this matters: the naive design — every broker synchronously confirms every metadata change with the controller — turns partition count into a coordination cost multiplier: 420 partitions means 420x the chatter on every leadership change. Treating metadata itself as a log that brokers tail asynchronously converts an O(n) synchronous coordination problem into an O(1) "check my cached version" read, which is exactly the same trick the data plane uses (append-only log + independent readers) applied to the control plane.
Scale-out trigger metric: sustained under-replicated partition count > 0 for more than 2 minutes, or controller request queue depth trending upward across 3 consecutive monitoring windows — both are leading indicators that current broker/controller capacity can't keep up with metadata or replication churn, well before producer-side latency (your segment-1 p99 target) actually breaches.
Why this metric and not raw CPU/disk: CPU and disk usage on a healthy cluster can look fine right up until the moment replication falls behind — under-replicated partitions is the metric that's actually causally close to your durability guarantee breaking, so it gives you lead time instead of a lagging confirmation that something already went wrong.
step 10
End-to-end assembly
Draw the whole thing from memory — both the hot/data path and the control/config path in one diagram.
✎ build it yourself first
Draw your complete system from memory: producer → broker → replication → consumer group → offset commit, and separately, the control-plane path: topic creation → controller → metadata propagation → client discovery. Do this before looking at the reference.
R13Reference: full assembly, both paths.reveal
DATA / HOT PATH
order-service --Produce(key=folio_id)--> Leader Broker (partition N)
--local append (fast)--> ack per acks-mode
--async replicate--> Follower Brokers --update ISR/high-watermark--
settlement-svc --Fetch(offset)--> Leader/Follower Broker --read log segment-->
--process record--> OffsetCommit(group, partition, offset) --stored durably--
CONTROL / CONFIG PATH
operator/CI --CreateTopic(name, partitions, RF)--> Controller Quorum
--allocate partitions across brokers--> write to internal metadata log
--brokers tail metadata log--> update local cache (leader/replica assignment)
producer/consumer client --DescribeCluster/Metadata()--> any broker
--returns cached leader/replica info--> client routes Produce/Fetch directly to leader
Both paths share one idea: the log pattern isn't just the product — it's the internal implementation strategy. Data replication is a log. Metadata propagation is a log. Even consumer group state changes are logged. Once you see that, "distributed log system" stops being a special-purpose messaging tool and starts being a general pattern for keeping many independent readers eventually consistent with one append-only source of truth — which is also, not coincidentally, the exact thing your segment-1 requirements asked for.
Final question — no answer key. Trace your own design, end to end, back against your own segment-1 FR/NFR list. Where's the gap? Is there a requirement you wrote on day one that your segment-6 algorithm or segment-8 state machine quietly doesn't actually satisfy? Find it yourself — that gap is worth more than anything in this document.